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1.0  AREAS  OF  INTEREST 

TNO  Human  Factors  distinguishes  the  following  areas  of  interest  as  regards  augmented,  mixed,  and 
virtual  environments  for  intuitive  human-system  interaction:  mixed  and  virtual  environment  operations, 
tele-operations,  command  &  control,  and  wearable  augmented  technology. 

2.0  CURRENT  RESEARCH  PROJECTS 

2.1  Mixed  and  Virtual  Environment  Operations 

The  goal  of  the  projects  in  this  area  is  realistic  simulation  and  training  of  flying,  driving,  and  dismounted 
soldier  team  operations  (MOUT),  and  virtual  prototyping  of  various  designs,  i.e.,  digital  models  of  ship 
bridges,  vehicles,  offices,  etc.  Various  projects  are  currently  carried  out  in  this  area,  using  the  facilities  and 
technologies  shown  in  Figures  1-5.  Special  attention  is  given  to  research  into  the  determinants  of  presence 
and  performance  in  mixed  and  virtual  environments,  focussing  on  team  operations  in  urban  terrain  and 
using  ‘agent’  and  ‘avatar’  digital  human  models. 


Figure  1:  Desdemona,  a  6DoF  Next  Generation  Motion  Platform  for  Spatial  Disorientation 
Training  and  for  Driving  and  Flight  Simulation  with  Sustained  G  Capabilities. 
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ORGANIZATION 


Figure  2:  Moving-Base  Driving  Simulator. 


Figure  3:  Military  Operations  in  Urban  Terrain  - 
Subject  immersed  through  head-mounted  display. 


Figure  4:  Ship  Bridge  Virtual  Prototyping. 
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Figure  5:  Manipulation  of  Objects. 


2.2  T  ele-Operations 

The  goal  of  the  projects  in  this  area  is  intuitive  control  of  remote  ground  and  air  vehicles  (for  tele¬ 
consultation:  interaction  of  distributed  subjects).  Various  projects  are  currently  carried  out  in  this  area, 
using  the  facilities  and  technologies  shown  in  Figures  6-10.  Special  attention  is  given  to  research  into 
the  determinants  of  presence  and  performance,  focussing  on  the  remote  ground  vehicle  (The  General’, 
Figure  6). 


Figure  6:  The  General’  Tele-Presence  Enabled  Unmanned  Ground  Vehicle 
(indoor  version;  at  the  left)  and  the  Ground  Control  Station  (at  the  right). 


Figure  7:  Unmanned  Aerial  Vehicle  (UAV)  Flying. 
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ORGANIZATION 


Figure  8:  Tele-Conferencing  -  VIRTUE  (Virtual  Team  User  Environment). 


Figure  9:  Tele-Maintenance.  At  the  left:  a  mechanic  at  work,  providing  a  picture 
to  the  expert  with  a  hand-held  camera.  At  the  right:  an  expert  guiding  the 
mechanic  at  a  distance,  communicating  with  video  and  duplex  audio. 


Figure  10:  Tele-Medicine  Body-Mounted  Equipment. 
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2.3  Command  &  Control 

The  goal  of  the  projects  in  this  area  is  to  develop  and  test  new  concepts  of  team  organisation,  human- 
system  task  integration  and  intelligent  interfaces  for  efficient  and  effective  command  and  control 
(example:  Figure  11).  Starting  with  a  set  of  missions  and  tasks,  the  trade-off  between  task  distribution, 
personnel  qualifications  and  support  concepts  is  analyzed.  Modeling  and  simulation  tools  are  used  to 
analyze  concepts  and  to  monitor  how  workload  is  distributed  over  the  personnel  during  a  scenario. 


Figure  11:  Mock-up  for  the  Landing  Platform  Dock  2  (Common  Operational  Picture). 


Research  on  intelligent  interfaces  focuses  on  situation  and  operator  state  dependent  adaptive  interface 
concepts  with  integrated  tools  for  information  management  and  decision  support.  Concepts  are 
implemented  and  tested  in  a  new  command  &  control  workstation  set-up  called  the  Basic-T  (Figure  12). 
It  incorporates  3D  perspective  and  stereo  images  of  the  tactical  picture. 


Figure  12:  Basic-T  (3D  Tactical  Picture). 


2.4  Wearable  Augmented  Technology 

The  goal  of  the  projects  in  this  area  is  to  provide  situation  awareness,  communication  and  navigation  for 
military  and  rescue  services  in  the  field.  The  deliverables  are  technology  demonstrators,  such  as  the  tactile 
suit  (Figure  13),  prototype  products,  such  as  integrated  headwear  (Figure  14),  and  specifications  (technical 
requirements  enlightening  human  factors  choices). 
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Figure  13:  Pilot  Wearing  a  Laboratory  Version  of  the  TNO  Tactile  Torso  Display  (TTTD),  or  Tactile 
Suit  (in  the  back:  a  RNLAF  Chinook  Helicopter).  This  particular  version  of  the  TTTD 
contains  twelve  columns  of  five  rows  of  vibro-tactile  actuators,  plus  four 
more  on  shoulders  and  between  the  upper  legs  and  the  seat. 


Figure  14:  Example  of  Gen.  3  Integrated  Head  Based  Systems  (courtesy  TNO). 


3.0  RESULTS  ACHIEVED  TO  DATE 

Publications  on  the  results  achieved  to  date  can  be  found  in  the  section  ‘Literature  prepared  by 
researchers’.  This  section  contains  brief  descriptions  as  regards  the  areas  distinguished. 

3.1  Mixed  and  Virtual  Environment  Operations 

Basic  research  has  been  conducted  on  spatial  orientation,  navigation,  and  glove  manipulation  by  subjects 
in  various  virtual  spaces  with  a  comparison  to  the  equivalent  real  space.  Virtual  prototyping  is 
implemented  as  a  flexible  and  user-collaborative  design  method.  The  method  has  been  applied  to  various 
workplaces,  such  as  ship  bridges,  control  rooms,  and  military  vehicles. 
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Desdemona  is  under  construction  and  is  scheduled  to  be  in  operation  at  the  end  of  2005. 

3.2  Tele-Operations 

In  the  past  a  range  of  human  factors  aspects  of  unmanned  (military)  vehicles  has  been  studied.  The  effort 
was  focused  on  remote  control  of  cameras,  dealing  with  such  issues  as  transmission  delays  and  bandwidth 
limitations,  situation  awareness,  and  image  augmentation.  In  more  recent  years,  issues  worked  on  have 
shifted  to  operator  training,  operator  skills,  ground  control  station  design,  automation  design,  human 
factors  guidelines  and  standards,  and  manned-unmanned  collaborations. 

3.2.1  UAV  Flying 

Various  interfaces  to  assist  the  operator  were  developed,  and  a  series  of  experiments  has  been  conducted. 
It  was  shown  that  the  manual  control  of  the  camera  of  an  unmanned  aerial  vehicle  (UAV)  can  be  difficult 
due  to  several  factors  such  as  time  delays  between  steering  input  and  changes  of  the  monitor  content,  low 
update  rates  of  the  camera  images,  and  lack  of  situation  awareness  due  to  the  remote  position  of  the 
operator  and  the  small  field  of  view  of  the  camera  images.  Furthermore,  if  the  UAV  is  controlled  from 
inside  an  aircraft  (moving  operator),  the  operator  has  to  deal  with  different  frames  of  reference. 

3.2.2  Tele-Presence  Control  Station 

A  new  track  is  the  work  on  tele-presence  control  stations.  Intuitive  (perceptual-motor)  control  of  remote 
camera  and  possibly  platform  will  free  up  the  cognitive  resources  that  are  needed  for  performing  the  task 
at  hand  (e.g.,  acting  in  the  remote  environment,  decision  making,  etc.).  ‘The  General’  is  the  first  attempt  to 
demonstrate  this  concept.  It  currently  features  immersive  head-slaved  stereo  images  and  3D  sound, 
vibration  sensations,  body  control  of  platform  movement,  and  remote  manipulator  control.  Special 
attention  is  given  to  research  into  the  determinants  of  presence  and  performance  in  tele-operation  settings. 

3.2.3  Tele-Conferencing 

VIRTUE  (virtual  team  user  environment)  aims  to  create  a  multiparty  meeting  space  where  virtual  and  real 
worlds  are  seamlessly  combined  so  that  participants  at  different  locations  can  have  the  impression  that 
they  are  sitting  next  to  each  other  and  can  work  co-operatively.  Experiments  were  carried  out  on  the 
effects  of  shared  background,  stereoscopic  vision  and  motion  parallax,  video  and  audio  delays,  and 
deviations  in  gaze  direction  on  tele-presence.  It  turned  out  that  tele-presence  was  not  influenced  by  these 
variables.  People  have  a  high  tolerance  toward  critical  factors  such  as  gaze  deviations  and  delays. 
However,  negative  effects  may  appear  under  more  critical  circumstances  and  with  more  sensitive  and 
more  objective  measures. 

3.2.4  Tele-Maintenance 

When  knowledge,  required  for  trouble-shooting  at  sea,  can  be  supplied  real-time  but  from  a  distance, 
problems,  such  as  with  the  limited  availability  of  specialists,  and  the  high  costs  of  maintenance,  may  be 
tackled.  An  experiment  was  performed  with  thirty  subjects,  in  which  an  on-shore  expert  and  an  on-board 
mechanic  fixed  technical  problems  together.  The  results  show  that,  in  principle,  the  problems  could  be 
fixed,  both  by  technical  and  non-technical  mechanics,  although  it  requires  more  time,  communication  and 
effort.  Unfortunately,  it  is  less  safe. 

3.2.5  Tele-Medicine 

The  tele-medicine  project  created  opportunities  to  study  an  on-line  mediated  workplace  involving  a  medic 
treating  a  patient,  and  a  doctor  at  a  remote  site  giving  advice.  Clinical  experiments  showed  the  effects  of 
equipment  and  settings  used,  the  mounting  of  the  equipment,  and  the  resulting  conversation.  A  direct 
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feedback  tele-viewing  system  addressed  the  issues  of  delays  and  decreased  image  quality  on 
telecommunication  links. 

3.3  Command  &  Control 

Various  new  command  concepts  have  been  created  and  tested  for  the  Dutch  armed  forces.  With  Basic-T  as 
a  testbed  the  different  tools  for  information  management  and  decision  support  have  been  tested  for  future 
application  in  the  maritime  environment. 

3.4  Wearable  Augmented  Technology 

3.4.1  Tactile  Suit 

The  tactile  suit  can  be  viewed  as  a  cockpit  instrument,  and  ongoing  research  is  establishing  its  value  for 
navigation,  hovering,  threat  warning,  spatial  disorientation  countermeasures,  communication,  and  other 
purposes.  The  same  technology  is  also  used  for  application  on  land  (navigation  support  and  threat 
warnings  for  drivers,  infantrymen,  blind  people,  etc.),  underwater  (divers),  and  in  space  (astronauts  in 
the  ISS). 

3.4.2  Integrated  Head  Based  Systems 

Future  soldiers  will  wear  helmets  in  various  degrees  of  encapsulation,  as  an  optimization  of  protection, 
perception,  and  usability.  Although  information  presentation  is  an  important  aspect  (new  type  of  non¬ 
obstructive  head  mounted  display,  high  resolution  on  the  move,  restoration  of  3D  environmental  sound 
while  wearing  hearing  protection),  the  human  factors  integration  is  key  to  head  based  systems  (design  for 
a  stable  fit,  weight  and  balance,  scalability  to  head  form,  anthropometric  variety,  hearing  protection, 
vision,  facial  recognition,  quick  conversion  from  grim  to  kind  appearance).  Head  based  systems  replace  a 
large  number  of  equipment  items  while  being  fully  compatible  with  the  soldier  system.  Head  based 
systems  are  a  platform  for  additional  functions  such  as  instrumented  vision,  antenna’s,  breathing 
protection  and  climatization.  Soldiers  need  for  instance  a  day  and  night  vision  capability,  that  allows  them 
to  supervise,  acquire  targets,  aim  and  shoot.  Speed  and  accuracy  compete  for  priority.  Shooting  comes  in 
direct  and  indirect  fire,  demanding  rather  different  aiming  techniques.  Though  nothing  is  virtual  at  this, 
it  is  an  outstanding  example  of  the  need  for  intuitive  human  system  interaction.  Procedures  need  not  be 
different  between  day  and  night,  sensor  images  need  to  combine  so  as  to  require  minimal  operation  for 
switching.  And  there  must  be  a  smooth  transition  between  searching  and  aiming,  without  losing  the  target. 
We  found  a  practical  solution  that  is  economical  and  convertible  to  various  military  tasks. 


4.0  COLLABORATIVE  PARTNERS 

TNO  Human  Factors  collaborates  with  a  large  variety  of  national  and  international  military  and  non¬ 
military  organisations.  Refer  to  www.tno.nl  for  more  information. 


5.0  LITERATURE  PREPARED  BY  RESEARCHERS 

This  section  contains  a  selection  of  the  literature  prepared  by  TNO-HF.  Refer  to  www.tno.nl  for  more 
information. 

5.1  Mixed  and  Virtual  Environment  Operations 

5.1.1  Basic  Research  and  Virtual  Prototyping 

Werkhoven,  P.J.  and  Groen,  J.  (1998).  Manipulation  performance  in  interactive  virtual  environments. 
Human  Factors,  40  (3),  pp.  432-442. 
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Bakker,  N.H.  (2001).  Spatial  Orientation  in  Virtual  Environments.  PhD  Thesis.  Delft,  The  Netherlands: 
Delft  University  Press. 

Delleman,  N.  and  Oudenhuijzen,  A.  (2003).  Advances  in  predicting  motor  behaviour,  comfort  and 
performance  for  digital  human  models.  Proc.  of  the  XVth  Triennial  Congress  of  the  International 
Ergonomics  Association  “Ergonomics  in  the  Digital  Age”,  August  24-29,  2003,  Seoul. 

5.1.2  Desdemona 

Bles,  W.,  Hosman,  R.J.A.W.,  and  De  Graaf,  B.  (2000).  Desdemona:  Advanced  Disorientation  Trainer  and 
(Sustained-G)  Flight  Simulator.  AIAA  Modeling  and  Simulation  Technologies  Conference.  Denver,  Co. 
August  14-17,  2000.  AIAA  2000-4176. 

Bles,  W.  (2001).  Desdemona:  Advanced  Disorientation  Trainer.  HSIAC  publication  Gateway,  SD-edition 
Fall  2001. 

5.2  T  ele-Operations 

5.2.1  UAV  Flying 

De  Vries,  S.C.  and  Jansen,  C.  (2002).  Situational  awareness  of  UAV  operators  onboard  of  moving 
platforms.  Proceedings  HCI-Aero  2002,  P  024. 

Van  Erp,  J.B.F.  and  Van  Breda,  L.  (2001).  UAV  Operations  using  Virtual  Environments.  What  is  essential 
for  virtual  reality  systems  to  meet  military  performance  goals?  AC/323(HFM-058)  TP/30. 
Neuilly-sur-Seine,  France:  RTO  NATO,  2001,  pp.  125-131,  2001  P  054. 

Veltman,  J.A.  and  Oving,  A.B.  (2003).  Augmenting  Camera  Images  for  Operators  of  Unmanned  Aerial 
Vehicles.  In:  The  role  of  Humans  in  Intelligent  and  Automated  systems.  (RTO-MP-088)  HFM 
symposium,  Warsaw,  Poland,  October  2002,  pp.  21.1-21.10. 

5.2.2  Tele-Conferencing 

Van  Besouw,  N.J.P.,  Van  der  Kleij,  R.,  Werkhoven,  P.J.,  and  Machin,  D.J.  (2000).  VIRTUE  -  virtual  team 
user  environment  requirements  specification  (Report  TM-00-D010).  Soesterberg,  The  Netherlands:  TNO 
Human  Factors. 

Schraagen  J.M.C.  (ed.)  Final  research  report  on  human  factors  experiments  (Report  TM-02-D006). 
Soesterberg,  The  Netherlands:  TNO  Human  Factors. 

Van  der  Kleij,  R.,  Paashuis,  R.M.,  Langefeld,  J.J.,  and  Schraagen,  J.M.C.  (in  press).  Effects  of  long-term 
use  of  video-communication  technologies  on  the  conversational  process.  Cognition,  Technology,  and 
Work. 


5.2.3  Tele-Maintenance 

Post,  W.M.,  Van  den  Boogaard,  S.A.A.,  and  Rasker,  P.C.  (2004).  Distributed  Trouble-Shooting. 
In:  Darses,  F.,  Simone,  C.  (Eds.)  Scenario-based  design  of  Collaborative  systems.  IOS  Press.  In  press. 

5.2.4  Tele-Medicine 

Hin,  A.J.S.  (1998).  Camera  configuration  and  communication  for  on-line  tele-consultation.  ITEC’98, 
Medical  Simulation  Conference. 
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Hin,  A.J.S.  and  Delleman,  N.J.  (2000).  On-line  teleconsultation:  a  human  factors  technology.  Proceedings 
of  ‘Combat  Medicine:  the  future  of  military  medicine  and  battlefield  support,  May  22-23,  2000, 
The  Hatton,  London.  London,  UK:  SMI. 

5.3  Command  &  Control 

5.3.1  Common  Operational  Picture 

Punte,  P.A.J.  and  Post,  W.M.  (2004).  Design  and  evaluation  Joint  Operations  Room  LPD-2.  Soesterberg, 
The  Netherlands:  TNO  Human  Factors.  In  press. 

5.3.2  Basic-T 

Van  Delft,  J.H.  and  Schraagen,  J.M.  New  MMI  concepts  for  situation  assessment  and  decision  making  in 
naval  command  and  control.  SCI-129  Symposium  on  ‘Critical  Design  Issues  for  the  Human-Machine 
Interface’,  Prague,  Czech  Republic,  19-21  May  2003. 

Van  Delft,  J.H.  and  Passenier,  P.O.  Interface  Concepts  for  Command  &  Control  Tasks.  NATO  RTA/HFM 
Symposium  on  ‘Usability  of  Information  in  Battle  Management  Operations’,  Oslo,  Norway,  10-13  April 
2000. 

5.4  Wearable  Augmented  Technology 
5.4.1  Tactile  Suit 

Van  Erp,  J.B.F.,  Veltman,  J.A.,  Van  Veen,  H.A.H.C.,  and  Oving  A.B.  (2003).  Tactile  torso  display  as 
countermeasure  to  reduce  night  vision  goggles  induced  drift.  NATO  RTO  meeting  on  ‘spatial 
disorientation  in  military  vehicles:  causes,  consequences  and  cures’  help  in  Spain,  15-17  April  2002, 
RTO-MP-086,  pp.  49.1-49.8. 


Van  Veen,  H.A.H.C.  and  Van  Erp,  J.B.F.  (2001).  Tactile  information  presentation  in  the  cockpit. 
In:  Brewster,  S.,  Murray- Smiths,  R.  (Eds.):  Haptic  Human-Computer  Interaction.  Springer  Verlag,  2001 
(Lecture  Notes  in  Computer  Science),  pp.  174-181. 

5.4.2  Integrated  Head  Based  Systems 

NATO  Standardization  Agreement  (STANAG)  Head  Borne  Systems  Standards  for  Dismounted  Soldier 
Systems,  first  draft  2004,  NATO  NAAG/TG1.  No  unclassified  references  available  at  this  time. 


6.0  VR  R&D  LABORATORY  FACILITIES  AVAILABLE 

Most  of  the  technology  used  can  be  found  in  the  new  SimLab  and  the  new  Desdemona  facility  (Figure  15). 
The  following  facilities  and  technology  are  available: 
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Figure  15:  TNO  Human  Factors  SimLab  and  Desdemona  Facility. 

6.1  Mixed  and  Virtual  Environment  Operations 

•  Visualisation  means  (HMD,  large  screen  projection,  3D-glasses) 

•  Moving  base  with  an  instrumented  car  or  truck  cabin 

•  Fixed  base  flight  simulator 

•  Desdemona,  a  6D0F  motion  platform 

•  Digital  human  models  (agents  and  avatars) 

•  Body  tracking  (locomotion,  gestures,  etc.) 

•  Data  glove  (manipulation  of  objects) 

•  3D  sound 

6.2  T  ele-Operations 

•  Ground  vehicle  operations  (‘The  General’) 

•  Unmanned  Aerial  Vehicle  flying 
Related  technology 

•  Tele-conferencing  /  virtual  team  user  environments 

•  Tele-medicine  (consultation  /  tele-knowledge) 

•  Tele-maintenance  (consultation  /  tele-knowledge) 

6.3  Command  &  Control 

•  Basic-T 

•  Common  Operational  Picture  mock-ups 

6.4  Wearable  Augmented  Technology 

•  Integrated  head  based  systems 

•  Tactile  suit 
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